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(54) Method of providing user interface 

(57) Apparatus, methods, and articles of manufac- 
ture consistent with the present invention provide a user 
interface for data processing systems wherein occlusion 
glyph code tilings (21 ) are used to facilitate the accurate 
extraction of user selection information from a substrate 
while presenting user Interface graphics to the user. The 
tiled code system in the present invention is comprised 
of tiles, with tiled code regions and occlusion regions (M 
Occlusion). In one embodiment, a first tiled code system 
facilitates robust decoding of the substrate with a suita- 
ble image capture. The occlusions are comprised of us- 
er interface graphics for user selection and infonnation, 



such as glyphs encoding a second code system. The 
occlusion regions occur periodically in each tile. The 
tiled code pattern can be used to implement a graphical 
user interface. A user uses a selection device to capture 
an image of a portion of the substrate that includes por- 
tions of tiled codes to complete a tile. The image is an- 
alyzed to determine the tile code pattern. A context code 
is determined from the tiled code pattern, and a selec- 
tion code Is detemiined from the occlusion infonnation 
in the Image. The context code and selection code are 
then processed by a syntax processor. Based on a se- 
lection or series of selections by a user, the syntax proc- 
essor sends commands to the system. 
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Description 

[0001] The present invention relates generally to a 
method of providing a user Interface for data processing 
systems. 

[0002] Codes can be embedded in an image on a sub- 
strate using machine-readable marks, such as glyph 
marks used in Xerox DATAGLYPH codes. The machine- 
readable marks can be captured in an image, and the 
image analyzed to extract information from the codes 
embedded In the image. Glyph marks In particular have 
several advantages. For example, glyph codes can be 
decoded from glyph marks even when the image carry- 
ing the glyph marks is distorted or there is some type of 
noise in the glyph marks. 

[0003] Another advantage of glyph marks is that they 
may have an unobtrusive visual appearance. If the glyph 
marks are sufficiently small, the glyphs appear as gray- 
scale to the unaided eye. For example, logically-or- 
dered, single-bit digital quanta may be encoded by re- 
spective elongated slash-like glyphs tilted to the left and 
right of vertical by approximately +45** and -45° for en- 
coding logical "Os" and "1s," respectively. The mutual 
orthogonality of the glyphs for the two logical states of 
0 and 1 of these single bit digital quanta enhances the 
discriminability of the code. Thus, the code pattern em- 
bedded in the glyphs can be recovered from an image 
of the glyphs, even when the code pattern is written on 
a fine grain pattern to cause the code pattern to have a 
generally uniform grayscale appearance. 
[0004] Decoding machine-readable marks, such as 
glyphs, in a substrate image requires an image capture 
window sized large enough to capture a sufficiently 
large set of glyphs from which the code pattern embed- 
ded therein can be decoded. If the capture area is too 
small, there will not be a sufficiently large set of glyph 
codes from which the embedded code pattern can be 
decoded. Therefore, the required size and shape of the 
image capture window will vary depending on the par- 
ticular code pattern embedded in the glyphs and the 
characteristics of the glyphs, such as size, shape, and 
orientation. For periodic tiled glyph patterns, an image 
capture window encompassing a tile provides an image 
area of glyphs that can be used for glyph decoding. If 
the capture window is shifted from alignment with a tile 
in a periodic tiled code pattern, the tile can still be re- 
constructed from the captured parts. See US-A- 
6,000,621. Once a complete tile has been captured, it 
can be decoded and the retrieved information used to 
perfomn a function. 

[0005] Unfortunately, prior art tiled code systems are 
somewhat limited in their use as graphical printing oc- 
clusions may obscure parts of a tile needed for decod- 
ing. What is needed Is a code system that enables use 
with graphical occlusions. It is also desirable that the 
code system accommodate additional code elements 
such as address codes to provide more information in 
addition to that found in prior art tiled code systems. 



[0006] It is sometimes additionally desirable to oc- 
clude areas of the substrate with graphical markings that 
may interfere with data glyph reading. For example, In 
a system implementing a graphical user interface on a 
substrate, graphics, such as icons, may occlude glyphs 
on the substrate. The user places an image capture de- 
vice over an area of the substrate to select an icon. The 
image captured by the image capture device typically 
includes regions where glyph codes are occluded. Con- 
ventional codes may not provide robust decoding be- 
cause they are not designed to compensate for the oc- 
clusions. 

[0007] What is needed is a code system that allows 
for occluded areas of glyphs and still provides a robust 
decoding of the underlying glyphs captured by an image 
capture device having a limited window capture size. 
[0008] Apparatus, methods, and articles of manufac- 
ture consistent with the present invention provide a user 
Interface for data processing systems wherein occlusion 
glyph code tilings are used to facilitate the accurate ex- 
traction of information from a substrate. The tiled code 
system in the present invention is comprised of tiles, and 
occlusions which may appear periodically or subperiod- 
ically across the tiles. The tiles are comprised of ma- 
chine-readable marks (e.g., glyphs) which can be de- 
coded, and the occlusions are comprised of information 
such as glyphs, graphics, or text which can also be de- 
coded. Decoding a tile code yields a context code and 
decoding an occlusion yields a selection code which is 
particular to the captured image. From the context code 
and selection code, the system is capable of determin- 
ing an action desired by the user, even when portions 
of the tiles are occluded. Additional objects and advan- 
tages of the invention will be set forth in part in the de- 
scription which follows, and in part will be clear from the 
description, or may be learned by practice of the inven- 
tion. The objects and advantages of the invention will 
be realized and attained by means of the elements and 
combinations particularly pointed out in the appended 
claims. It is to be understood that both the foregoing 
general description and the following detailed descrip- 
tion are exemplary and explanatory only and are not re- 
strictive of the invention, as claimed. 
[0009] The accompanying drawings, which are incor- 
porated in and constitute a part of this specification, il- 
lustrate an embodiment of the invention and, together 
with the description, serve to explain the principles of 
the invention. 

[0010] Fig. 1 illustrates an overview of the properties 
of a glyph code pattern; 

[0011] Fig. 2 illustrates a glyph code pattern having 
repeating rectangular glyph tiles; 
[0012] Fig. 3 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with periodic fixed 
marked occlusions; 

[0013] Fig. 4 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with periodic variable 
marked occlusions; 
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[0014] Fig. 5 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with periodic occlu- 
sions with variable marking including overlay glyph 
codes; 

[0015] Fig. 6 illustrates a glyph code pattern having s 
repeating rectangular glyph tiles with occlusions subpe- 
riodic In the X direction; 

[0016] Fig. 7 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with occlusions subpe- 
riodic in the X and Y direction; 
[0017] Fig. 8 Illustrates subperiodic strip occlusion 
rectangular glyph tiling and tile capture window as an 
application to text print GUI; 

[0018] Fig. 9 shows embedded data in the form of an 
array of rectangular tiles in alternating shades of gray 
scale to illustrate one embodiment rectangulartile struc- 
tures; 

[0019] Fig. 10 illustrates the embedded data of Fig. 9 
without gray shading; 

[0020] Fig. 11 is a flowchart illustrating writing of a 
glyph code pattern having repeating rectangular glyph 
tiles with occlusions; 

[0021] Fig. 12 is a flowchart illustrating reading and 
decoding of a glyph code pattern having repeating rec- 
tangular glyph tiles with occlusions; 
[0022] Fig. 13 illustrates an address carpet code; 
[0023] Fig. 1 4 shows the address carpet code of Fig. 
13, filling the occlusion regions of tiled code of columns 
of glyphs indicated In bold; 

[0024] Fig. 1 5 illustrates an embodiment using a tiled 
code and an occlusion code as part of a user selection 
process; and, 

[0025] Fig. 1 6 shows an image capture system that 
may be used to select an Icon and capture a correspond- 
ing portion of embedded data. 

[0026] Methods, apparatus, and articles of manufac- 
ture disclosed herein consistent with the principles of 
the Invention implement a code pattern tiled for a win- 
dow capture having multiple occlusions of the glyph pat- 
tern in which the code pattem is embedded. 
[0027] Fig. 1 illustrates properties of glyph marks and 
codes embedded in the glyph marks. Glyph mari<s are 
typically implemented as a pattern on a substrate, such 
as glyph marks 21 on substrate 24. Glyph marks 21 are 
not easily resolved by the unaided human eye. Thus, 
glyph marks 21 appear to the unaided eye to have a 
unifonn gray scale appearance or texture. 
[0028] Enlarged area 23 shows an area of glyph 
marks 21 . Glyph marks 21 are typically comprised of 
elongated slash-like marks, such as glyph 22, and are 
typically distributed evenly widthwise and lengthwise on 
a lattice of glyph center points to fomn a rectangular pat- 
tern of glyphs. In one embodiment, glyphs are tilted 
backward or fonward, representing the binary values of 
"0" or "1 respectively. For example, glyphs may be tilt- 
ed at +45° or -45° with respect to the longitudinal dimen- 
sion of substrate 24. Using these binary properties, the 
glyph marks can be used to create a pattern of glyph 
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marks representing O's and Vs embodying a particular 
coding system. 

[0029] The glyph marks of enlarged area 23 can be 
read by an image capture device and decoded into O's 
and 1 's by a decoding device. Decoding the glyphs into 
O's and Vs creates a glyph code pattern 25. The O's and 
1's of glyph code pattern 25 can be further decoded in 
accordance with the particular coding system used to 
create glyph pattern 21. Additional processing might be 
necessary to resolve ambiguities created by distorted 
or erased glyphs. In one embodiment, glyph code pat- 
tern 21 comprises tiles of glyph marks. A tile is a shaped 
area, such as a rectangle, which can be repeated to cov- 
er the substrate without overiap. Consistent with the 
principles of the present invention, a tile is required to 
be used as the basis for a periodic tiling. This is analo- 
gous to the tiling of a wall with rectangular bricks. Tile 
shapes other than rectangular may also be used. 
[0030] In one embodiment, each tile comprises an 
identical pattern of glyph marks encoding identical infor- 
mation. Forexample, the tile infomnation encoded by the 
glyph marks in each tile might comprise a label identify- 
ing the substrate, in addition to other infonriation about 
the substrate, tiles, or glyph marks on the substrate. 
Thus, information can be derived by decoding the tiles. 
For example, in one embodiment the decoded informa- 
tion defines characteristics of the underlying substrate 
and the glyph mari<s on the substrate. 
[0031] Additionally, the codes in the tiles of glyph 
marks have particular properties that allow predeter- 
mined occlusions of the tiles without compromising de- 
codability of the information encoded in a tile of glyph 
marks. An occlusion is an area of a tile that does not 
necessarily have the same coding scheme as the re- 
mainder of the tile. For example, an area of a tile might 
be obliterated by an icon graphic printed over glyph 
marks of the tile or some glyph lattice positions may be 
written with glyphs of another coding system, which may 
not necessarily be identical from tile to tile. 
[0032] Each tile may have one or more predefined oc- 
clusion shapes in predefined respective locations in the 
tile. For example, the occlusions may be laid out with 
tiling vectors that are an integer fraction, such as one- 
half or one-third of the repeating contiguous tiles of 
glyph mari<s in one or two directions. 
[0033] The occlusions may also have variable Images 
within congruent occlusion tile shapes. Forexample, the 
occlusions may have different mari<s and colors. The oc- 
clusions may additionally comprise different visual indi- 
cia, such as graphics, text, glyphs, or a combination of 
different types of visual indicia. An occlusion that in- 
cludes glyphs could be decoded to provide infomnation. 
For example, the decoded infonnation may provide in- 
fomnation relevant to the particular occlusion image 
such as identification or type. 

[0034] Fig. 2 illustrates a glyph code pattern having 
repeating rectangular glyph tiles. Tiling vectors T^ and 
Ty define a rectangular glyph tile comprised of glyph 
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codes C1 1 to C99. The glyph tile regularly repeats In the 
horizontal and vertical directions to fornn a tiling which 
is periodic in each direction. Glyph codes C11 to C99 of 
each tile comprise a glyph code pattern representing in- 
fomnation, such as a label identifying the substrate on 
which the glyph tiies appear. 

[0035] Fig. 2 also illustrates a tile-shaped capture win- 
dow, denoted by a broken line surrounding a rectangular 
area of the glyph code defined by corners C88, C87, 
C78 and C77. Capturing an innage of an area encom- 
passing at least the shape and size of a tile will include 
an instance of each common glyph mark of the tile. Be- 
cause the captured image includes each common glyph 
mark of a tile, the tiled code can be reconstructed from 
the captured image. 

[0036] Fig. 3 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with periodic fixed 
marking occlusions. The occlusions are periodic in that 
they appear with regular frequency across the array of 
tiles. In Fig. 3, each tile has an occlusion ("M occlusion") 
covering a particular region within the tile. As shown in 
Fig. 3, each occlusion covers code positions C27, C28, 
C29, C37, C38, C39, C47, C48, and C49 in each tile. 
[0037] Fig. 4 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with periodic variably- 
marked occlusions. As in Fig. 3, the occlusions are pe- 
riodic in that they appear with regular frequency across 
the array of tiles. In the embodiment of Fig. 4, however, 
the occlusion in each tile ("Mxx occlusion") may have a 
unique marking. For example, as shown in Fig. 4, each 
occlusion may be a unique graphic icon. Using icons is 
particularly useful in implementing a graphical user in- 
terface. 

[0038] Fig. 5 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with periodic occlu- 
sions that include glyph codes. Fig. 5 is similar to Fig. 4 
in that each tile has an occlusion covering particular 
glyph lattice positions within the tile. Each occlusion 
marking ("Mxx occlusion") may be unique. In Fig. 5, 
however, the occlusions include glyphs that change the 
visually apparent shape of the occlusion. These glyphs 
may be decodable to provide a code, such as a code 
identifying a particular occlusion. Occlusion glyphs may 
also be merely decorative. For example, occlusion M12 
in the upper middle tile includes glyph codes S29, S39, 
and 549, and occlusion M22 (not labeled in Fig. 5) in the 
lower middle tile includes glyph codes G27, G28, G29, 
G37, G38, G39, G47, G48, and G49. Thus, occlusions 
M12 and M22 include glyph codes on the same lattice 
as the tiled glyph codes, but have a code system S and 
G that may be different than tiled code system C. In an- 
other example, the glyph codes in the occlusion area 
comprise address codes spanning multiple tiles and 
may be used to aid in reconstruction of the tiles from a 
captured image. 

[0039] Fig. 6 illustrates a glyph code pattern having 
repeating rectangular glyph tiles with subperiodic occlu- 
sions in the X direction. Each tile includes two occlu- 



sions, thus creating subperiodic occlusions in the X di- 
rection. These occlusions may include glyphs, as illus- 
trated by occlusion M14 and occlusion M24. 
[0040] Fig. 7 illustrates a glyph code pattern having 

5 repeating rectangular glyph tiles with subperiodic occlu- 
sions in the X and Y direction. Each tile includes four 
occlusions, thus creating subperiodic occlusions in the 
X and Y direction. Additionally, the occlusions may in- 
clude glyphs, as illustrated by occlusion M14. 

10 [0041] Fig. 8 illustrates a glyph code pattern having 
repeating rectangular glyph tiles fomning strip occlu- 
sions. The strips are formed from contiguous occlusions 
in the X direction. In the embodiment shown in Fig. 8, 
text is marked on occlusions in the glyph pattern of re- 

15 peating tiles. 

[0042] Fig. 9 shows embedded data in the form of an 
array of rectangular tiles in alternating shades of gray 
scale to illustrate one embodiment of rectangular tile 
structures. The center of each tile as shown in Fig. 9 

20 may have an occlusion. In Fig. 9, the occlusion of the 
tile at the center is a document icon. The occlusion may 
also comprise other graphics, text or glyph codes. When 
glyphs fill in the occluded area, the occlusion glyphs can 
be visually indistinquishable from the tiled code. 

25 [0043] Fig. 1 0 illustrates the embedded data of Fig. 9 
without gray shading. The occlusionsf illed in with glyphs 
are visually indistinguishable from the tile glyphs. 
[0044] Fig. 11 is a flowchart illustrating writing of a 
glyph code pattern having repeating rectangular glyph 

30 tiles with occlusions. A bitmap image of tiles with occlu- 
sions is created (step 224), and the bitmap image is 
printed on a substrate (step 226). In another embodi- 
ment, the tiles are printed on a substrate and the occlu- 
sions are printed over the tiles in a second printing op- 

35 oration. In yet another embodiment, all glyphs are print- 
ed together on a substrate and non-glyph occlusion 
graphics are printed over or under the glyphs in a sec- 
ond printing step. This last embodiment facilitates better 
alignment of all the glyphs. 

40 [0045] Fig. 12 is a flowchart illustrating the process 
perfomned by a processor when reading and decoding 
a glyph code pattern having glyph tiles with occlusions. 
In one embodiment, the process is performed by a giyph 
reading device coupled to a processor programmed to 

45 perfomri the steps depicted in the flowchart of Fig. 12. 
The process could also be perfomaed by special pur- 
pose hardware or a combination of hardware and soft- 
ware. 

[0046] A reading device is used to capture an image 
50 of an area of glyph marks from a substrate (step 124). 
The glyphs in the captured image are converted into a 
matrix of O's and 1's (step 126). The matrix is decoded 
to identify code portions of the tile codes and their logical 
addresses (step 127). Based on the code portions and 
55 their logical addresses, the tiling vectors are detennined 
(step 128), from which the tile is logically reconstructed 
(step 129). The tile is then decoded, and the occlusion 
in the tile may be processed. The occlusion is decoded 
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as well If the occlusion includes decodable information, 
such as glyphs (step 130). Another example of decoda- 
ble occlusion infomnation is a mark in a check box which 
may be decoded using existing mark recognition meth- 
ods. Using the decoded tiled information and the infor- 5 
mation from the decoded occlusion, If any, processing 
continues by perfomning an operation associated with 
decoded tile infonnation and the decoded occlusion in- 
formation (step 132). 

[0047] Fig. 13 illustrates an address carpet code. 
[0048] Fig. 14 shows the address carpet code of Fig. 
13 filling the occlusion regions of tiled code in columns 
of glyphs indicated in bold. The tiled code period in the 
X direction includes four columns spanning tiling vector 
T1 . The address carpet codes can be used to establish 
the logical order or address of the tiled code pattern. 
[0049] Capture window A illustrates a capture area of 
sufficient extent so as to encompass an area from which 
the underlying address code and tiled code pattern can 
be decoded. In the embodiment Illustrated in Fig. 14, the 
area encompasses at least a length T1 in the x tiling 
direction and a height in the y direction so that confident 
values of n consecutive bits in each of address codes 
u, V, and tiled code d can be decoded con-ectly. The pre- 
cise location of the capture window may vary. For ex- 
ample, capture window B illustrates a capture area hav- 
ing the same shape and area as capture window A, but 
shifted in the x and y directions relative to capture win- 
dow A. 

[0050] In another embodiment of the coding system 
of Fig. 14, graphics are included on the substrate with 
the tiled pattern and address carpet codes. The graphics 
may be comprised of glyphtones, or may partially ob- 
scure the tiled pattern and/or the address carpet codes. 
The graphic must be of such a character so as not to 
obscure the tiled pattern or address carpet codes to the 
point where they cannot be decoded. For example, if the 
graphic obscured a sufficiently large number of glyphs 
which define the tiled pattern, the pattern could not be 
decoded from a captured image. 
[0051 ] Fig. 1 5 illustrates an embodiment using a tiled 
code and a selection code as part of a user selection 
process. A context code is decoded from the tiled code 
pattern, and an address pattern located within occlusion 
regions provides a selection code by spatially locating 
the selection. Additional selection codes may be provid- 
ed by angular orientation of the capture and user control 
signals via buttons etc. The context code and selection 
code are provided to syntax processing 404. Syntax 
processing 404 detennines whether an action should be 
taken or whether additional selections need to be made, 
as is conventional in user selection processing. 
[0052] This two-level coding system can be used in 
several ways, such as to implement a graphical user in- 
terface. To use this user interface, a user places an im- 
age capture device on or near a selection, such as a 
graphic which may form an occlusion. An image is cap- 
tured which includes an area large enough to allow the 



tile pattern to be decoded. The image is then decoded 
to determine the context code from the tile pattern and 
a selection code associated with the position of selec- 
tion. This allows tiled codes to be used in a system for 
capturing an image, decoding the image to determine 
one or more particular codes from the image, and per- 
fonning operations based on the one or more particular 
codes. 

[0053] Fig. 1 6 Is a block diagram of an image capture 
and processing system consistent with the invention 
that may be used to: capture a user-selected portion of 
a substrate 432 comprising glyph marks 434 (shown 
generally as light gray print) and occlusions 436 (shown 
generally as dark gray print); decode the embedded da- 
ta markings in the captured portion; and perfonn further 
processing on the decoded information, as described 
herein. In one embodiment, computer system 412 is a 
general purpose computer system, such as a conven- 
tional personal computer or laptop computer, that in- 
cludes main memory 416, read only memory (ROM) 
418, storage device 420, processor 422, and communi- 
cation interface 424, all Interconnected by bus 426. Bus 
426 also connects to display 430, cursor control 414, 
and frame capture 428. 

[0054] An image capture device, which in this case is 
a camera pen 41 0 or camera mouse 431 , is connected 
to frame capture 428. Either camera pen 41 0 or camera 
mouse 431 may be used, but camera pen 410 will be 
used for purposes of discussion. Camera pen 410 cap- 
tures an image from substrate 432 and transfers the 
captured image to frame capture 428. The user makes 
a selection by placing camera pen 41 0 on or near visual 
indicia on glyph marks 432, and pressing one or more 
buttons on the device. For example, pressing button 41 4 
causes camera pen 410 to capture the portion of the 
substrate 432 under the tip of camera pen 410, and 
transmit the image to computer 412, via frame capture 
428, for analysis. The depression of one or more buttons 
can be used for additional signaling, as in a double click, 
hold down. 

[0055] As shown in Fig. 16, substrate 432 is com- 
prised of embedded data, such as glyph marks 434 and 
occlusions 436. Frame capture 428 collects the cap- 
tured data and separates it into Individual frames. These 
results, together with signals from cursor control 41 4 are 
transferred to processor 422 for further processing. In 
one embodiment, camera pen 41 0 also transmits button 
depression signals to frame capture 428 and cursor con- 
trol 414. The signals cause frame capture 428, cursor 
control 414, and processor 422 to process the images 
and button depression signals. 
[0056] Processor 422 processes the results from cur- 
sor control 414 and the captured frame image from 
frame capture 428 as previously discussed in respect to 
Fig. 12. In one embodiment, processor 422 computes a 
context code, a location code, and a rotation code from 
the data. The context code defines a particular context 
of substrate 432, such as a substrate page number. The 
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location code defines the location on substrate 432 from 
location where the captured image was taken. Using the 
context code location code and rotation code (optional- 
ly), processor 422 accesses information related to the 
context code and location code. For example, if the con- 5 
text code represents substrate page iv, location code in- 
dicates location coordinates x=97 and y=92, and rota- 
tion = 43^ then processor 422 would retrieve infonna- 
tion corresponding to that specific location and rotation 
from substrate 432. 

[0057] In one embodiment, the retrieved information 
includes two types of infonnation: a bit map image rep- 
resentation of substrate page iv, and icon information 
defining the locations and meaning of icons on substrate 
page iv. In operation, a user selects an icon representing 
a system object on substrate 432 via an image capture 
device. In response, a system attached to the computer 
system of Fig. 16 (not shown) can use the retrieved 
meaning of the icon to perform functions corresponding 
to the icon, as a conventional system would in response 
to user selection of an icon on a graphical user interface 
in a conventional display. 

[0058] In one embodiment, main memory 41 6 is a ran- 
dom access memory (RAM) or a dynamic storage de- 
vice that stores instructions executed by processor 422. 
Main memory 416 may also store information used in 
executing instructions. ROM 41 8 is used for storing stat- 
ic information and instructions used by processor 422. 
Storage device 420, such as a magnetic or optical disk, 
also stores instructions and data used in the operation 
of computer system 412. 

[0059] Display 430 may be a CRT or other type of dis- 
play device. Cursor control 414 controls cursor move- 
ment on display 430. Cursor control 414 may accept sig- 
nals from any type of input device such as a trackball, 
or cursor direction keys. 

[0060] The system shown in Fig. 16 can be used to 
capture and decode the occluded embedded data de- 
scribed herein. Thesystem may also implement the two- 
level syntax processing discussed with respect to Fig. 
15. The apparatus and methods described herein may 
be implemented by computer system 412 using hard- 
ware, software, or a combination of hardware and soft- 
ware. For example, the apparatus and methods de- 
scribed herein may be implemented as a program in any 
one or more of main memory 41 6, ROM 41 8, or storage 
device 420. In one embodiment, processor 422 exe- 
cutes programs which analyze captured portions of oc- 
cluded embedded data to determine address informa- 
tion encoded in the embedded data. 
[0061] Such programs may be read into main memory 
416 from another computer readable medium, such as 
storage device 420. Execution of sequences of instruc- 
tions contained in main memory 416 causes processor 
422 to perform the process steps any other physical me- 
dium with patterns of holes, a RAM, a PROM, an 
EPROM, a FLASH-EPROM, any other memory chip or 
cartridge, a carrier wave as described hereinafter, or any 



other medium from which a computer can read and use. 
[0062] Various fomns of computer readable media 
may be involved in canying one or more sequences of 
instructions to processor 422 for execution. For exam- 
ple, the instnjctions may initially be carried on a mag- 
netic disk or a remote computer. The remote computer 
can load the instructions into its dynamic memory and 
send the instructions over a telephone line using a mo- 
dem. A modem local to computer system 412 can re- 
ceive the data on the telephone line and use an infrared 
transmitter to convert the data to an infrared signal. An 
infrared detector coupled to appropriate circuitry can re- 
ceive the data carried in the infrared signal and place 
the data on bus 426. Bus 426 carries the data to main 
memory 416, from which processor 422 retrieves and 
executes the instructions. The instructions received by 
main memory 416 may optionally be stored on storage 
device 420 either before or after execution by processor 
422. 

[0063] Computer system 41 2 also includes a commu- 
nication interface 424 coupled to bus 426. Communica- 
tion interface 424 provides two way communications to 
other systems. For example, communication interface 
424 may be an integrated services digital network (IS- 
DN) card or a modem to provide a data communication 
connection to a con-esponding type of telephone line. 
Communication may also be, for example, a local area 
network (LAN) card to provide communication to a LAN. 
Communication interface 424 may also be a wireless 
card for implementing wireless communication between 
computer system 41 2 and wireless systems. I n any such 
implementation, communication interface 424 sends 
and receives electrical, electromagnetic or optical sig- 
nals that carry data streams representing various types 
of infomnation. 

[0064] The link between communication interface 424 
and external devices and systems typically provides da- 
ta communication through one or more networks or oth- 
er devices. For example, the link may provide a connec- 
tion to a local network (not shown) to a host computer 
or to data equipment operated by an Internet Service 
Provider (ISP). An ISP provides data communication 
services through the world wide packet data communi- 
cations network now commonly referred to as the Inter- 
net. Local networks and the Internet both use electrical, 
electromagnetic or optical signals that carry digital data 
streams. The signals through the various networks and 
the signals between the networks and communication 
interface 424, which carry the digital data to and from 
computer system 412, are exemplary forms of carrier 
waves transporting the information. 
[0065] Computer system 412 can send messages 
and receive data, including program code, through the 
network(s) via the link between communication inter- 
face 424 and the external systems and devices. In the 
Internet, for example, a server might transmit a request- 
ed code for an application program through the Internet, 
an ISP, a local network, and communication interface 



15 



20 



25 



30 



35 



40 



45 



50 



6 



11 



EP1 215 617 A2 



12 



424. 

[0066] Program code received over the network may 
be executed by processor 422 as it is received, and/or 
stored in memory, such as in storage device 420, for 
later execution. In this manner, computer system 412 
may obtain application code in the form of a carrier 
wave. 

[0067] The term "computer-readable medium" as 
used herein refers to any medium that participates in 
providing instructions to processor 422 for execution. 
Such a medium may take many fomris, including but not 
limited to, non-volatile memory media, volatile memory 
media, and transmission media. Non-volatile memory 
media includes, for example, optical or magnetic disks, 
such as storage device 420. Volatile memory media in- 
cludes RAM, such as main memory 416. Transmission 
media includes coaxial cables, copper wire and fiber op- 
tics, including the wires that comprise bus 426. Trans- 
mission media can also take the fomn of acoustic or light 
waves, such as those generated during radio wave and 
infrared data communications. 

[0068] Various forms of computer readable media 
may be involved In carrying one or more sequences of 
instructions to processor 422 for execution. For exam- 
ple, the instructions may initially be carried on a mag- 
netic disk or a remote computer. The remote computer 
can load the instmctions into its dynamic memory and 
send the instructions over a telephone line using a mo- 
dem, A modem local to computer system 412 can re- 
ceive the data on the telephone line and use an infrared 
transmitter to convert the data to an infrared signal. An 
infrared detector coupled to appropriate circuitry can re- 
ceive the data carried in the infrared signal and place 
the data on bus 426. Bus 426 candies the data to main 
memory 416, from which processor 422 retrieves and 
executes the instructions. The instructions received by 
main memory 416 may optionally be stored on storage 
device 420 either before or after execution by processor 
422. 

[0069] Computer system 41 2 also includes a commu- 
nication interface 424 coupled to bus 426. Communica- 
tion interface 424 provides two way communications to 
other systems. For example, communication interface 
424 may be an integrated services digital network (IS- 
DN) card or a modem to provide a data communication 
connection to a corresponding type of telephone line. 
Communication may also be, for example, a local area 
network (LAN) card to provide communication to a LAN. 
Communication interface 424 may also be a wireless 
card for implementing wireless communication between 
computer system 41 2 and wireless systems. I n any such 
implementation, communication interface 424 sends 
and receives electrical, electromagnetic or optical sig- 
nals that carry data streams representing various types 
of information. 

[0070] The link between communication interface 424 
and external devices and systems typically provides da- 
ta communication through one or more networks or oth- 



er devices. For example, the link may provide a connec- 
tion to a local network (not shown) to a host computer 
or to data equipment operated by an Internet Service 
Provider (ISP). An ISP provides data communication 

5 services through the worid wide packet data communi- 
cations network now commonly referred to as the Inter- 
net. Local networks and the Internet both use electrical, 
electromagnetic or optical signals that carry digital data 
streams. The signals through the various networks and 

10 the signals between the networks and communication 
interface 424, which carry the digital data to and from 
computer system 412, are exemplary forms of carrier 
waves transporting the information. 
[0071] Computer system 412 can send messages 

15 and receive data, including program code, through the 
network(s) via the link between communication Inter- 
face 424 and the external systems and devices. In the 
Internet, for example, a server might transmit a request- 
ed code for an application program through the Internet, 

20 an ISP, a local network, and communication interface 
424. 

[0072] Program code received over the network may 
be executed by processor 422 as it is received, and/or 
stored in memory, such as in storage device 420, for 
25 later execution. In this manner, computer system 412 
may obtain application code in the fomn of a carrier 
wave. 

30 Claims 

1 . A method of providing a user interface with embed- 
ded data, comprising: 

35 embedding a tiled code pattern to encode a 

context code; and 

embedding an occlusion region on each tile, 
where the occlusion region comprises user in- 
terface graphics. 

40 

2. The method of claim 1, wherein the occlusion region 
comprises a selection information. 

3. The method of claim 1 or claim 2, wherein the em- 
45 bedded data comprises glyph codes. 

4. The method of any of the preceding claims, further 
comprising capturing the embedded data. 

50 5. The method of claim 4, further comprising the step 
of decoding the captured data. 

6. The method of any of the preceding claims, further 
comprising the step of processing the syntax of the 

55 context code and the selection code. 

7. An embedded data code on a substrate, compris- 
ing: 



7 
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periodic tiles, each comprising: 

a predefined code area having a context 
code pattern comnnon to the othertiles; and 
a predefined occlusion having a graphical 5 
selection infonnation pattern. 

8. The embedded data code of claim 7, wherein the 
embedded data comprises glyphs. 

10 

9. The embedded data code of claim 7 or claim 8, 
wherein the occlusion comprises a graphic, ad- 
dress codes, text or glyphs. 
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(54) Method of providing user interface 

(57) Apparatus, methods, and articles of manufac- 
ture consistent with the present invention provide a user 
interface for data processing systems wherein occlusion 
glyph code tilings (21) are used to facilitate the accurate 
extraction of user selection information from a substrate 
while presenting user interface graphics to the user. The 
tiled code system in the present invention is comprised 
of tiles, with tiled code regions and occlusion regions (M 
Occlusion). In one embodiment, a first tiled code system 
facilitates robust decoding of the substrate with a suita- 
ble image capture. The occlusions are comprised of us- 
er interface graphics for user selection and infonnation, 



such as glyphs encoding a second code system. The 
occlusion regions occur periodically in each tile. The 
tiled code pattem can be used to implement a graphical 
user interface. A user uses a selection device to capture 
an image of a portion of the substrate that includes por- 
tions of tiled codes to complete a tile. The image is an- 
alyzed to detenninethe tile code pattem. A context code 
is detemnined from the tiled code pattern, and a selec- 
tion code is detennined from the occlusion information 
in the image. The context code and selection code are 
then processed by a syntax processor. Based on a se- 
lection or series of selections by a user, the syntax proc- 
essor sends commands to the system. 
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